Papers with Shannon entropy
Estimating the Entropy of Linguistic Distributions (2022.acl-short)
Copied to clipboard
| Challenge: | Shannon entropy is a quantity of interest for linguists studying the communicative capacity of human language. |
| Approach: | They propose to use entropy estimators to estimate linguistic effects from observed data . they propose to recommend better entropic estimators for future linguistic studies . |
| Outcome: | The proposed estimators are over-estimated due to poor entropy estimators, the authors argue . they recommend the same estimators be used in future linguistic studies. |
Efficient Computation of Expectations under Spanning Tree Distributions (2021.tacl-1)
Copied to clipboard
| Challenge: | a general framework for inference in spanning tree models is presented . a spanning-tree model is a combinatorial structure that is important for natural language processing . |
| Approach: | They propose a general framework for inference in spanning tree models . they exploit a connection between expectations and gradients, which allows efficient algorithms . |
| Outcome: | The proposed framework is up to 15 times faster than existing algorithms and reduces runtime complexity by a factor of the sentence length. |
Large-Scale Bitext Corpora Provide New Evidence for Cognitive Representations of Spatial Terms (2024.eacl-long)
Copied to clipboard
| Challenge: | Recent evidence suggests that there exist two classes of cognitive representations within the spatial terms of a language. |
| Approach: | They propose a pipeline for extracting, isolating, and aligning spatial terms from parallel text . they find evidence that variability in functional terms differs significantly from that of geometric terms . |
| Outcome: | The proposed pipeline extracts, isolates, and aligns spatial terms in basic locative constructions from parallel text. |
Tokenization and the Noiseless Channel (2023.acl-long)
Copied to clipboard
| Challenge: | Subword tokenization is a key part of most NLP pipelines, but little is known about why some combinations lead to improved downstream model performance. |
| Approach: | They propose that good tokenizers lead to efficient channel usage . they propose that an optimal encoding assigns extremely long codes to low-frequency subwords . |
| Outcome: | The proposed tokenizers have a very strong correlation with BLEU in machine translation . the proposed function can be used to improve model performance in the downstream task . |